Use of an Hourglass Model in Neuronal 

Coding. 
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We study a system of interacting renewal processes which is a 
model for neuronal activity. We show that the system possesses an 
exponentially large number (with respect to the number of neurons in 
the network) of limiting configurations of the " firing neurons" . These 
we call patterns. Furthermore, under certain conditions of symmetry 
we find an algorithm to control limiting patterns by means of the 
connection parameters. 
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1 Introduction 

Definition of the model. We study here the hourglass model. This model 
is represented by a system of interacting renewal processes, numerated by 
the sites in a finite subset Ac Z". Define for any z G A its neighbourhood: 



*Postal address: SAMOS, Universite Paris 1, 90, rue de Tolbiac, F-75634 Paris Cedex 
13, France 

t Postal address: University of Lund, Department of Mathematical Statistics, Box 118, 
S-221 00 Lund, Sweden 



Abstract 





1 



where {Ik, k = 1, . . . , K} C Z u is a fixed but arbitrary set of non-zero vectors. 
Notice by our definition, that z D(z), and for any z,y G A 

^ G £>(y) iffy G £>(z). 

Also let for any subset J C A 

D(J) := {j±4,£; = l,...,X, j G J}. 

Define on some probability space S, P) some random variables Aj(0), 
Yi, 9ij, i G A, j G Z?(i), to be independent with densities having a certain 
number of moments. The variables Aj(0) and Yi, i G A, are assumed to be 
positive and to represent the initial state and the self-characteristic of the ith 
neuron, respectively. We shall call the 6^- the connection parameters. For any 
fixed i G A, j G D(i), the distribution of is assumed to be concentrated 
either on R_ or R + . We will specify the sign of in every example we treat 
below. Assume also % = for any % G A, j G" D{i). Finally let Y} n) , Y^ z) 
and Of, 9^' z \ n > 1, z G A, be independent copies of the variables Y { and 

respectively. 

We shall define a Markov process X(t) = (Xi(t), % G A), t > 0, with 
left-continuous trajectories in R+ as follows. For alH G A and t > define 

X i (t)=X i (Q)-t + £ Y ^ (1-2) 

0<t n <t: Xi(t„)=0 



+ E E ((^ (nJ) - < of} 

jeD(i) 0<t n <t: Xj(t„)=0 

-efi{Xi(t n ) > $>}) 



-EE E <>T n - 

jeD(D(i))\{i}uD(i) zeD(i)nDU):9 zi <o 0<t n <t: Xj(t n ) = 0, 

Xi(t n ) > Of, and X z (t n ) < Oft* 

Biological interpretation. Any moment t such that X z {t) = for some z G 
A we call the moment of firing of the zth neuron. Assume for a moment that 
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6ji = for any i, j G A. Then the last two summation terms in the right-hand 
side of equation (1.2) turn into zero, meaning that Xi(t) is merely a renewal 
process in this case. Thus each component X^t) represents the duration 
of time before the next firing of the corresponding ith neuron assuming no 
interaction takes place meanwhile. 

In the presence of non-zero interactions the dynamics of X(t) = (Xj(t), 
i G A), t > 0, are described as follows. As long as all the components of X(t) 
are strictly positive, they decrease from the initial state X(0) linearly in time 
with rate one until the first time t Zl that one of the components reaches zero 
for some z\ G A: X Zl (t Zl ) = 0. We say that at this moment t Zl , the Z\th 
neuron fires and sends impulse 6^ to the jth neuron, if j G D(zi). This 
means the following. At the moment t zi , the trajectory X zi (t zi ) jumps to a 
random value i.e. 

X Zl (t Zl +) = Y}p, (1.3) 

which corresponds to the first summation term in (1.2). At the same moment 
every trajectory Xj(t) with j G D(zi) receives an increment 9^ independent 
of the other processes, more precisely, according to the second summation 
term in (1.2) 

f X 3 (t Zl )-9^, X Xj(t n ) - 6% > 0, 
Xj(t n +) = i (1.4) 
[ Y*> 1 \ otherwise. 

Notice, that by (1.4) the negative connections 6^ delay the moment when 
Xj(t) hits zero, i.e. the moment of firing of the j-th neuron. Whereas positive 
connections can shorten the time-interval until the next firing. That is why 
we shall call negative connections inhibitory and the positive ones excitatory. 

Case 1. Assume that all the interactions 0^-, j G D(i), are negative. 
Then definition (1.2) simply becomes 

xr\t) = xm-t+ E ^ (n) +E E 1^1,(1.5) 

0<t„<t: Xi(t„)=0 jeD(i) 0<t n <t: Xj(t„)=0 

for i 6 A and t > 0. We use a notation X tnh {t) for this particular case. This 
model is due to Cottrell (1992) [1] and it has been intensively studied (see 
an account of the previous results in Section 2 below). 
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Case 2. Assume, some of the interactions 6ij are positive. Notice that, 
only if 9 zi j 1 > 0, it may happen that Xj 1 (t zi ) — < 0. In this case 

the trajectory X^it) is reset instantaneously to an independent value 
according to (1.4). Furthermore, we say that the jith neuron also fires at the 
moment t zi and changes instanteneously the trajectories of the neighbouring 
neurons according to the last term in (1.2), which means the following. 

Given X Zl (t Zl ) = 0, define the set of the firing at the same moment t Zl 
neurons: 

F 1 (z 1 ,t Zl ) := {ji G D( Zl ) : X h {t Zl ) - 9^ < 0}. 
Then at time (t zi +) the state of the system is defined as follows: 

X 3 (t Zl +) (1.6) 



= < 



Y?\ if jE{z 1 }UF 1 , 



Xj(t Zl ) - Ei Off, if j e £>(Fi) \ u F 1 U , 

where Fi = F 1 (2; 1 ,t Zl ), the summation Y] i runs over the set {i & Fi : i G 
D(j) and < 0}, and any summation over an empty set equals zero. The 
rest of the trajectories Xi(t) with % g D(Fi) U Fi U D(z\) U {^i} remain 
unchanged. After moment t Zl the foregoing dynamics are repeated. 

To end this description recall the well-known fact from physiology that a 
neuron does not react to the incoming impulses during a certain period right 
after it's own firing. This period is called a refractory period. Observe that 
in our model a neuron receives an impulse only in the moments when it does 
not fire itself. This reflects the property of the refractory period. 

This type of neural network has been proved to be equivalent, in a sense, 
to the "classical" neural model, which describes the interacting membrane 
potentials (see [12]). 

Plan of the paper. Our paper is organized as follows. In Section 2 
we find the critical values of the parameters which separate the ergodic and 
transient cases. Also we provide some historical comments on our model in 
Section 2. In Section 3 we analyze a fully connected network with inhibitory 
connections only, in which case we solve a problem related to the memory 
capacity. 
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2 Critical parameters. 
2.1 Results. 

Here we formulate our results on the critical values of the parameters which 
separate the ergodic and transient cases of the networks with excitatory and 
inhibitory connections. 

In order to eliminate boundary effects, we shall assume here that A = 
{—N,...,N} U is a //-dimensional torus, i.e. we identify any two points 
(i!, ...,i v ) and (J u . . .,j v ) in Z u whenever \i m - j m \ G {0, 2N} for every 
1 < m < v. Hence, any point in A has the same number of neighbours. 
Further we assume that N > 1 and v G N + are fixed but arbitrary (although 
only the cases v — 1, 2, 3 are relevant for our context). Define for any i,j G A 

V 

¥ - J II A := W -Jk\2N, 
k=l 

where \x — y\2N '■— min{|x — y\, \x — y — 2N\, \x — y + 2N\} for any x, y G Z. 
Let A be the subset of the points in A such that: 

i) the origin (0, . . . , 0) G A , 
ii) \\x — y || a 7^ 1 for any x, y G A . 

Consider now the network X(t) = (Xi(t),i G A) defined in (1.2) with the 
following connection architecture. 

Assumption 1 Let the neighbourhood D{i) defined in (1.1) be such that 

D{i) = D I (t)UD E {t), 

where 
and 



with \Dj(i)\ 
but arbitrary 
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D I (i) = {je A: ||i-j|| A = l} 

(2.1) 



D E (i) CA\A zffteA\A , 
2v and \D E (i)\ = K E . The constants < K E < N are fixed 



Then we set 

-wml 3 , ifjeDj(i), 



Oij - < 



(2.2) 



where 77^, i 7^ j are independent copies of positive independent random vari- 
ables i] k , respectively, k = 1, 2, with F,r] k = 1, and where wi and we are pos- 
itive parameters of the inhibitory and excitatory connections, respectively. 

In words the condition (2.2) means that the nearest connections are in- 
hibitory while the more distant ones are excitatory. 

Assume further that X(0), Xi(0),i e A, are i.i.d., and also Y, Y i: i e A, 
are i.i.d. with EF = 1. 

Assumption 2 Assume, that the distributions ofY,r)i, 772, and X(0) /iove 
densities go(u) , gi(u) , g2(u) andgs(u), respectively, which are positive differ- 
entiable functions, such that for some positive constants a and a 

gk{u)<ae~ au for all u > and k = 0,1,2,3. (2.3) 

We note that the assumption of the exponential decay, though seemingly 
rather restrictive, arises naturally from the physiology (see, for example [12] 
and the references therein). 

Theorem 1 Under Assumptions 1 and 2 for any we > there exists a 
positive wY{we) such that the system X(t) with parameters (2.2) is transient 
if 

Wl > w?(w E ), (2.4) 

and ergodic if 

Wl < wf{w E ). (2.5) 

We specify the critical function wf{w E ) in (2.15) below. Generally speak- 
ing, this function depends on N . However, its asymptotic behaviour when 
w E — * is uniform in N, as we establish in the following theorem. 

Theorem 2 There exist positive constants C and Cq independent of N such 
that 

I wT(w E ) - (±- - ^w E ) |< Cw% (2.6) 



2u 2v 



for all < we < Cq. 
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We postpone the proofs of these theorems to the next section. 

So far only the case we = has been studied analytically. Cottrell 
[1] proved that when w E = 0, the network X(t) = X inh {t) (see (1.5) ) is 
ergodic, whenever wi < l/(2z/), which is, clearly a particular case of (2.6). 
In [1] sufficient conditions for convergence and transience were found for the 
finite network X(t) with Ac Z 2 and a specific structure of the connections. 
Further Piat [10] (1994) extended these results to a more general connection 
structure. Karpelevich et al. (1995) provided a complete analysis of the 
evolution of inhibitory networks when the matrix of the expected values of 
the connections defines a self-adjoint operator. 

Paper [2] presents simulated results for the model, in which both in- 
hibitory and excitatory connections were incorporated. Here for the first 
time we study analytically a rigorous mathematical model for such a net- 
work. Our result explains the slope of the diagram given on Fig. 4 [2] in the 
neighbourhood of the critical value (0, l/(2z/)). The most attractive feature 
of the hourglass network for neuromodelling is that in the transient case the 
system splits as t j oo into two subsets. These are a subset of active (i.e. 
infinitely often firing) neurons and one of inactive neurons, which can be 
recognized as dark and white areas, respectively, in the simulations, see e.g. 
figures 3 and 6 of [2] . A rigorous definition of the possible limiting patterns of 
active and inactive neurons, called traps (see Section 2.2.1 below) was given 
by Karpelevich, Malyshev and Rybko in [7]. Thus the set of all possible traps 
for a network should be naturally thought of as a system of patterns, which 
the network can hold, that is, memorize and recognize, hopefully. Recently 
[8] obtained a result on phase transitions in the thermodynamical limit, which 
establishes the large memory capacity of these networks. 

Further (in Section 3) besides a description of the possible patterns, we 
solve an inverse problem. This is the problem of how to determine the con- 
nections in order to get a system which stores a given system of patterns. 

2.2 Proofs of Theorem 1 and Theorem 2. 
2.2.1 Preliminary definitions and results. 

We shall use in our proofs the results of [7]. Therefore firstly we introduce 
the terminology of [7] for our model. 
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For any non-empty W C A we call X w (t) the restriction of process X(t) 
on the set W. This is defined as X(t), but with initial conditions 

w | X,(0), ineW, 

* v 7 [ oo, if i ^ W, 

in which case we assume that A^(£) = oo if 2 ^ W, for all t > 0, i.e. the 
nodes A \ are " deleted" . 

Assume, the process (X^(t), i e W) is ergodic for some non-empty W C 
A. In this case we define for any i <E W and j G De{i) H the following 
limiting frequencies: 

Trf' := lim ^#{0 <t n <T : X™(t n ) = 0}, (2.7) 

and 

7r^ e := hm i#{0 <t n <T : Xf (t n ) = and (t„) - df < 0}. (2.8) 

Thus 7Tj W '° is the limiting frequency of firing of the ith neuron due to the 
hitting by the trajectory X™(t), while n^' e is the limiting frequency of 
firing of the ith neuron due to an immediate excitatory impulse from the jth 
neuron. Then it is natural to call the (total) limiting frequency of firing of 
the ith neuron the following sum of the defined above limits: 

*Y ■= «r + E <^ i e W. (2.9) 

j€D E (i)nw 

Next we define the second vector field v w = (v^ , j £ A\W) by the following 
formula for its components: 



vf = -l- Y, - E E^-Trf' , jeA\W. (2.10) 

ieD I (j)nw ieD E (j)nw 



Formally vY is the limiting mean drift of the j'th component Xj^(t). Indeed, 
according to our definition (1.2) the ith neuron sends inhibitory impulses to 
the corresponding neighbours with the limiting frequency iif ', but it sends 
excitatory impulses with the limiting frequency iif' ■ For the details on the 
application of the theory of the second vector field we refer to [7] and the 
references therein. 
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We shall call a trap for the process X(t) any non-empty set M C A such 
that the process (X^ M (£), i e A\ M) is ergodic, while any coordinate of the 
second vector field v A \ M is positive, i.e. 

vf W > for all j G M. (2.11) 

For future reference let us rewrite now in our notations the criteria from 
[7] on inductive ergodicity and transience conditions. 

Theorem A (See Theorem 2.1, [7].) I. Inductive ergodicity. If for any 
W C A the process (X^(t),i G W) is ergodic and 

vf < for all j G A \ W 

then the process is ergodic. 

II. Sufficient transient conditions. If there exists a set M which is a trap for 
the process X(t), then the process X(t) is transient. 

2.2.2 Subsystems with excitatory connections only. 

Consider now the restriction X A °(t). Notice that due to the definition 
of A and Assumption 1 the only connections between the components of 
(X A °(t),i G A ) are excitatory. Then by the definition of the process X A °(t) 
we have for all % G A : 

X Ao (t)=X i (0)-t+ ]T F (n) (2.12) 

0<s„<t: X A °(s n )=0 

- E E (w EV ^ ) i{x A °(t n )>w EV ^ ) } 

jeD E (i) 0<tn< t: xf"(t n )=0 

+(X A °(t n ) - Y^)I{X A »{t n ) < w E ^ j) }) , 

where r)^ 1 '^, n > 1, j G A , are independent copies of the variable r} 2 - Clearly, 
for any % G Ao the one-dimensional x\ % ^ (t) is ergodic. Further for any w E > 
and for any W C A such that (X™(t),i G W) is ergodic, the second vector 
field defined in (2.10) has negative coordinates only: 

vf = -l- E ^e^°, jeA \W. 

ieD E (j)nw 
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Hence, by Theorem A on inductive ergodicity {X i °(t),i G Ao) is ergodic for 
any fixed w E > 0. Taking into account translation invariance, we derive from 
(2.12) and (2.9) 

Trfo = : 7r + (w E ) = ir + '°(w E ) + K E n + ' e (w E ) > (2.13) 

for any i G A . In particular, 

7r+(0) = 7T + '°(0) = 1/EY = 1, (2.14) 

since in the case w E = we have a system of independent renewal processes 
numerated by the sites of A . 

Thus one can view the system X(t) as a result of inhibitory interactions 
between two (independent at initial moment) excitatory networks: (X^t), 
i G A ) and (X^t), i G A \ A ). 



2.2.3 Proof of Theorem 1. 

We shall show that under condition (2.4), with 

wf(w E ) = ^ r (2.15) 

the set A is a trap for the system X(t). Then the first statement on tran- 
sience in our theorem follows from Theorem A. 

As we have shown above, (X A °(t), i G A ) is ergodic. Computing the 
second vector field with respect to formula (2.10), and taking into account 
(2.2) and (2.13) we get 

v t° = - 1 + E w ^t° = -1 + 2v Wl ir + (w E ), j G A \ A . (2.16) 

Hence condition (2.4) together with (2.15) is equivalent to 

vf°>0, jGA\A , (2.17) 

i.e. inequality in (2.11) is fulfilled for all j G A \ A , which implies that A is 
a trap. Hence, our statement on transience follows by Theorem A. 

Assume now that condition (2.5) with (2.15) holds. Clearly, for any i G A 
one-dimensional x} l \t) is ergodic. Next we will show that for any ergodic 
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face W the components of the second vector field v w defined by (2.10) are 
negative. It obviously follows from our definition of the model, that the 
limiting firing frequency is the highest when the neuron has the excitatory 
connections only and the maximal number of them, i.e. 

for any W G A, any i G A \ W and j G A \ Ao From here and (2.10) we 
derive for any i G A \ W 

v w < _i + 2vw I n + {w E ) < 0, 

where the last inequality is due to (2.5). Hence, using Theorem A (Theorem 
2.1 [7]) on inductive ergodicity, we readily derive our statement on ergodicity. 
Theorem 1 is proved. 



2.2.4 Proof of Theorem 2. 

Let pA{t,UA), ua G R+' A ', denote for any finite A C A the density of the 
process X%>(t) = (X^"(t),i G A). In the particular case we = 0, we denote 
the corresponding density by p° A (t, ua)- Due to ergodicity the following limit 

lim p A (t, u A ) =: Pa{u a ) (2.18) 

t^oo 

exists for any A C A . In particular, we have for the limiting density of an 
independent renewal process: 

lim p A (t,u A ) =P°aM = IbSK) ( 2 - 19 ) 

where 

Further we will use the following lemma stating that the limiting density 
in (2.18) is a small perturbation of the function in (2.19) when the parameter 
of interaction we is sufficiently small. 

Lemma 1 There exist positive and independent of N constants C, j3 and c 
such that for any < we < c and for any subset A C A 

\p A (t,u A )-p° A (t,u A )\ < w E C^e-^eA^ % (2.21) 

for any t > and u A G R^ 1 . 
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Proof. Recall that go and gs are the density functions of the variables 
Y and AT(0), respectively. Let pit, u A(V v a ), u Ao ,v Aq G R+ A °, denote the 
transition density of the process (X A °(t), i G A ). Then for any A C A the 
density PA(t, ua), ua G R+' A ', is given by the formula 

PA(t,u A )= / ,.,,,/ . II 9z{u z ) \ p(t,u Ao ,v Ao ) du Ao dv A(M . (2.22) 

In the particular case we = 0, we denote the transition density by 
p°(t, u Ao , v Ao ). Clearly, in this case 

p°(t,u Ao ,v Ao ) = ]J P°(t, u j, v j) 

iGA 

due to the independence of the components. Notice, that the transition 
density of each of the independent renewal processes is 

p°(t,u,v) = (2.23) 



j S(u-t-v), if < t < u, 

" j 9o(t - u + v) + Efcli lt U Ps k ( x )9o(t - u + v - x) dx, if < u < t, 

for every u,v G R + and t > 0, where <5(m — •) is the Dirac distribution 
concentrated at u; Y,Y k , k > 1, are iic?.; S k := X)f=i ^ > 1, and p Sk is 
the density of the distribution Sk- 

Using the results of Stone [11] one can show that under Assumption 2 
there exist positive constants C and a' such that 

\p°(t + t', 0, v) - p°(t, 0,v)\ < C'e- a ' (t+v ^ (2.24) 

for all v G R+ and t,t' > (for the details see for example, [13] Section 4). 
Combining (2.24) and (2.19), we get: 

\p°(v)-p°(t,0,v)\ < C'e~ a ' {t+V) . (2.25) 

(For the general theory of piece-wise linear stochastic processes we refer to 
the book [3] by Davis, 1993.) 

After these preliminaries the proof of Lemma 1 follows by standard tech- 
niques using so-called cluster expansions (see, e.g. [13] for a similar model 
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and [6] for a class of stochastic processes with local weak interactions) as 
soon as the Kolmogorov equation (see (2.26) below) is written down for the 
transition density p(t, u Ao , v Ao ). 

Let M(R+°) denote a class of densities /x on R+° which possess all con- 
tinuous partial derivatives. Define also 



p(t,u Ao ,T) = J^p(t,u Ao ,v Ao )dv Ao 

for any Borel set T G R+°. Denote here 0(i) = Dgii) and % = e. Then 
for our model we derive for any density /i G M(R+°) and for any Borel set 

r g R+° 

d f 

di /r a » ^ UAo ' p(yt ' UA °' ^ dUAo ^ 2 ' 2 ^ 



r ^ Q 

J n A S M M A ) Q^P(t,UA ,T)du Ao 



d 

igA 

+ J R Ao M m A ) EiGAo ^(^) / R i_+2|O(0l PY(Wi) (l\jeO(i)PY(yj)Pe V2 (W: 
X [p(*,«Ao + U A ( u Mv w i[j 0(i)>yO(z)),T) - p(t,U Ao ,T)\ 

x dy (€) dWiijofl du Ao , 
where S(-) is the Dirac measure concentrated at 0, 

Py(u) = g (u), p em {u) = | f^ 6 ^ e ' if e = M ~ °' 
and the vector U Ao (u Ao , u^y yo(i)) has components: 

Ujiu^w^o^yoii)) 

' w t , if j = i, 

= < —WjI{iVj < uj} + (yj — Uj)I{wj > Uj}, if j G 0(i), 
0, otherwise. 

Further for any Wi > let the vector Wi = {wi j,j G A ) have the following 
components: 

*}■■ = [ Wu if 3 = h 
lJ | 0, otherwise. 
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Let us rewrite now formula (2.26) in the following operator form: 

d f 

di /r a o /i ^ Ao ' ) p(yt ' MAo ' ^ dUAo ^ 2 ' 27 ' ) 

= L »(uA ) E f-^-p(*,«Ao,r) 

+ 5(ui) p Y (wi)[p(t,u Ao +Wi,T) - p(t,u Ao ,T)}dwi j ofo Ao 



+ J R A M«A<>) Ei 6 A / R i+ 2 | o(o | py(tWi) (njeO(i)Pr(%)p67j : 

x dj/ 0(i) d^(j 0(i) du Ao 



='■ / A . A*(«Ao) J2( H o A ° + Hl' Ao )p(t,u Ao ,T)du Ao . 
Jn + ieA 

Notice, that when e = the operator if^' A ° = for any i. In this case (2.27) 
simply becomes 



d_ 

di 



/ Ao M«A„) P°(t,U Ao ,T)du Ao = / ^(uao) E ^' A >°(t,MA ,r)dM Ao 

JR + JR + ieA 

(2.28) 

i.e., the equation for the density of the process whose components are in- 
depended renewal processes. Hence the formula (2.23) gives a solution to 
(2.28): 

p°(t,u Ao ,T) = p°(t,u Ao ,v Ao )dv Ao . 

Now we can find the solution to equation (2.27) by solving the equivalent 
integral equation (for the reference see also [13] and [6]): 

/ Ao ^i(u Ao )p(t,u Ao ,T)du Ao = fi{u Ao )p {t,u Ao ,T)du Ao (2.29) 

J R._i_ J R i 
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> m Ao> m a ) H i P( s ' M A ' r ) du\ Q du Ao ds. 

JO JR + JR + . gAo 

Finally from (2.29) we obtain the formula for the density defined by (2.22): 

p A (t,v A )=p A (t,v A ) (2.30) 

oo ft f f 

fc=l (zi,..., Jfc )6Ag + + 

H\ kM p°(s k , u k Ao ,v Ao ) dv Ao \ A du\ o ds k ... du\ o du Ao ds 1: 

Making use of the formula (2.23) and the bounds (2.3) and (2.24), one can 
prove the convergence of the series in (2.30) and get the necessary bounds 
by following the the formulae in Section 3 of [13] (p. 179 and further) with 
only minor modifications due to (2.26). Therefore for the sake of brevity we 
skip the rest of the straightforward part of the proof of Lemma 1. 

Consider now the embedded Markov chain (xi(n) := X^ (r n ),i G A ), 
n — 1,2,..., where t±, T2, . . . denote the consecutive moments when at least 
one of the components of X A ° reaches zero, i.e. Xf°(r n ) = for some j G Aq. 



Due to ergodicity of (X^°(t), i G A ), there exists a limiting distribution, call 
it F, of x(n) as n — > oo. Let x°°(n) be the stationary version of the Markov 
chain x(n), i.e. whose initial distribution is F. Further for any % G Ao and 
j G De(i) let Zij be a random variable with distribution function 

Fij(u) := P{x?(n) < u \ xf(n) = 0}. 
Clearly, the density function f\j for each F^ is given by 

/«(«) = 5# (™) 

as long as pj(0) > 0. Observe that p°(0) = ^ = 1 by (2.20). Hence, 
according to Lemma 1 condition Pj(0) > is satisfied at least for all small 
values of we- 
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We shall derive now an equation for definition (2.7) 

and (2.13) ). From (2.12) and the above arguments on ergodicity we obtain 
for all % G A : 

lim -X 4 Ao (t) = = -1 + n + '°(w E )EY - K E n + '°{w E )w E (2.32) 

t— >oo t 

- ^ +, °(w E )E(Z ij — Y — w E r] 2 )I{Z i:j < w E r) 2 }, 

jeD E (i) 

where Z^, Y and i] 2 are independent by their definitions. Therefore taking 
into account that EF = 1, we obtain from (2.32) the following equation: 

7r + '°(w E ) I 1 - + X! < «W 

V jGD E (i) 

+ ]T E(w E ?72 - Z ij )I{Z ij < w E T] 2 }\ = 1. (2.33) 

ieD B (i) / 

Lemma 2 There exists a positive constant C\ independent of N such that 
for any i G A and for any j G D E {i) 

\P{Zij < w E r] 2 } - w E \ < w 2 E d (2.34) 

and 

E(w E r] 2 - Z^IiZij < w E r] 2 } < w 2 E d (2.35) 
for all < w E < c (with c defined in Lemma 1). 

Proof of Lemma 2. In the case w E = let = Zfj. Then the density of 
Zfj (call it correspondingly ffj(u) ) is given by 

f%(u)= P \u)= / g (y)dy (2.36) 

due to (2.31) and (2.20). 

First let us obtain the following bound as a corollary of Lemma 1: 

\fM-fij(u)\ <w E C 2 , u G R+, (2.37) 
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where C 2 is some positive constant independent of N. Notice, that from 
(2.21) and (2.18) we immediately derive: 

\pa(ua) -Pa( u a)\ < w E C 3 exp{-f3^2ui}, (2.38) 

for any ua G R+, where C3 is some positive constant independent of N. 
Then the bound (2.37) readily follows from (2.38) and the continuity of the 
transformation (2.31). 

Let us prove now (2.34). Consider 

roc fWEV 

P{Z ij <w E r} 2 } = J o g 2 (y) J q f%(u)du dy (2.39) 

roc rw E y 

+ j o 92(y) I (/«(«) - $■(«)) du dy. 

Substituting formula (2.36) into the first integral in (2.39) we derive after 
simple calculations: 

roc rw E y n 

I 92(y) I f^{u)dudy (2.40) 

roo 

= / P{V2 > y}w E (1 - P{Y < w E y}) dy 
Jo 

roo 

= w E -w E / P{i] 2 > y}P{Y < w E y} dy, 
Jo 

where we used the condition E^ 2 = 1. Taking into account (2.3) it is easy to 
see that 

w E / P{t7 2 > y}P{Y < w E y} dy < C'w 2 E (2.41) 
Jo 

for some positive constant C. Next, making use of (2.37) we readily derive 

roc rWED 

1 1 92(y) J o (/«(«) - /gH) du dy\ < w 2 E C 2 . (2.42) 

Combining the bounds (2.42) and (2.41) together with equations (2.40) and 
(2.39) gives us (2.34). 

To prove (2.35) let us consider the following decomposition: 

E(w E r) 2 - Z i3 )l{Z ij < W E T) 2 } 

roo / rw E y \ 

= J o 92{y)yJ^ (w E y-u)f i:j (u)duJ dy 
17 



roc / rw E y \ 

+ J o 92{y) (J {w E y - u){f i:j {u) - fijiu)) duj dy. 

Using again (2.3) and (2.37) we immediately derive from here: 

E(w E r} 2 - Z id )I{Z l3 < WEV2} < w 2 E C 3 + w 3 E C 4 , (2.43) 

where C3 and C4 are some positive constants, which implies (2.35). The 
lemma is proved. 

Lemma 2 allows us to derive from equation (2.33) that 

\ir + '°{w E ) - 1\ <w 2 E C 5 (2.44) 

for some positive constant C5. 

Finally let us consider np e (w E ) for i e A ,j e D E (i). According to 
definition (2.8) we derive similarly (2.32) 

itf e = lim i#{0 < t n < t : Xf{t n ) = and X^(t n ) - < 0} (2.45) 
= hmi £ I{X^(t n )<w EV i n) } 

0<t„<t: X 3 A °(t n )=0 

= 7T + <°(^) P{% < U^Jfc}. 

Substituting (2.45) into definition (2.13) we obtain 

n + (w E ) =n + >°(w E )(l+ Yl P{Zij<w E r] 2 }), (2.46) 

jeD E (i) 

which together with (2.44) and (2.34) gives us the following bound: 

\n + (w E ) - (1 + K E w E )\ < w 2 E C 6 (2.47) 

for all w E < Co, where C§ and Cq are some positive constants. 

Substituting the bound (2.47) into (2.15) we get (2.6). This completes 
the proof of Theorem 2. 
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3 Fully connected finite networks. 



3.1 The model. 

Let A = {1,...,N}. Consider an N- neuron network with inhibitory con- 
nections only, i.e. X(t) = X mh (t) as defined in (1.5). Here we assume that 
D(i) = A\{i} for any i G A, which means that our network is fully connected. 

Let N = 2pk, where k,p G Z + are fixed arbitrary. Suppose a network 
consists of 2p symmetric subnets (blocks), determined by the values of the 
connection constants E#y as follows. We divide set A = {1,...,N} into 2p 
subsets Wi, i = 1, ...,2p, so that 

\Wi\ = k and U^i W, L = A. (3.1) 

Notice that we keep k > 1 fixed but arbitrary. This is simply a size of one 
block or unit of our network. The case k — 1 is trivial. 

Let us divide also the set {1, 2p} into a set of p non- intersecting pairs, 

{l,...,2p} = U p n=1 C n = U£ =1 {C*,0, (3.2) 



where 

Then we set 



\C n \ = 2, and Q D Cj = 0. 



EY X =a> 0, 

E6 xy = -c^ < 0, if x e W u y G Wj, 
for all i,|/6 A and 1 < i, j ' < 2p, where 



(3.3) 



= | c, if {i,j} = C n for some n, . ^ 

lj \ b, otherwise, 

for some arbitrary but fixed constants 

< b < a < c. (3.5) 

Thus our network consists of 2p connected blocks of interacting identical 
neurons. The connections between the neurons of different blocks are dif- 
ferent in general from those between the neurons of the same block. In the 
following Theorem 3 below, we describe all the possible traps for our net- 
work, i.e. all the possible limiting states in the transient case. Notice that 
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the total number of the patterns is \T\ = 2 P = 2^, which is exponentially 
large with respect to N. This description will allow us to find a method 
to reconstruct or "memorize", in some particular cases, a given system of 
traps (see Corollary 1 below). More precisely, we will show how to choose 
the appropriate connection constants. 

3.2 Description of patterns. 

Theorem 3 Let N = 2pk, where k > l,p > 1, and decompositions (3.1) 
and (3.2) be fixed but arbitrary. Call C the set of all the decompositions of 
{1, 2p} into a pair of subsets such that 

C : = {(Lx, L 2 ) : L 1 U L 2 = {1, 2p}, L 1 DL 2 = 0, (3.6) 

LjnC„ = l, i = 1,2, n = 1, 

Then a set A C A is a trap for the network with the connection constants 
(3. 3)- (3. 5), if and only if 

A = {x e A : x G U i£B Wi}, (3.7) 

for some B C A such that 

(B,{l,...,2p}\B)eC. (3.8) 

Proof. First we will show that any set A satisfying conditions (3.7) and 

(3.8) is a trap for the above defined system. Indeed, consider a restriction 
X A \ A (t). In this case (X^ A (t), i G A\ A) is a completely connected system, 
and for any x,y E A \ A, y ^ x 

-W xy = b < a = EY X , (3.9) 

which implies ergodicity of (X^ A (t), i G A \ A) due to Proposition 2.2 in 
[7]- 

Assume, (X^it), i G W) is ergodic for some W C A. Observe that in the 
case of a fully connected network with only negative connections definition 

(2.9) becomes 

vrf = vrf , i G W. (3.10) 
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Furthermore it follows by ergodicity from (1.5) and definition (2.7) that 

nf = (ey, + ]T E|^ ,ieW. (3.11) 

V jeW\{%} J 

Correspondingly, we derive from (2.10) 

V J = -1 + E E|^.|7rf , j G A \ W. (3.12) 

Notice that definitions (3.1)-(3.2) imply \B\ = p for the set B satisfying 
conditions (3.7)-(3.8). Then for any x G A \ A we derive from (3.11) and 
condition (3.4) (see also [7]), that 

= I = I 131 

x a + (|A\A|-l)6 a + (pk-l)b' { ' 

Substituting (3.13) into definition (3.12) and taking into account (3.3)-(3.4), 
we obtain for any y G A 

a\a 1 ck+{p-l)bk 

v y -=~ l+ a + ( P k-i)b> (3 - 14) 

which together with (3.5) implies 

v^ A > 0. 

Hence, we conclude that A is a trap. 

Next we will show that any trap satisfies (3.7)-(3.8). Clearly, any subset 
AC A can be represented as 

A = {x G A : x G U ieB Ai}, (3.15) 

where B C {1, 2p} and Ai C W; L for any i G B. Suppose a subset A does 
not satisfy conditions (3.7)-(3.8), which happens if and only if at least one 
of the following situations (I) or (II) takes place: 
(I) the set B does not satisfy (3.8), which means that 

B = {C n , n G /} U {C l n , neJ} (3.16) 
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and 

B :={!,..., 2p}\B = {C n , ne J'}U{Cj, n E J}, (3.17) 
where {1,1'} = {1,2} and sets 7 and I' are such that 

7 U /' ^ 0, J n /' = 0, and / U /' U J = {1, . . . , 2p}; (3.18) 

(II) Wi \ Ai 7^ at least for some i E B, i.e. (3.7) is not satisfied. 

Suppose situation (I) takes place. Without loss of generality let / = 1 
and /' = 2. Consider 

A := A \ A = {U ieB WA U {U ieB Wi \ A t }. (3.19) 

Define the subset B x C 5 so that 

WiXAi^OiffieBi, (3.20) 

and thus 

^4 = {U i€ BWi} U {U ieBl Wi \ ^}. 
Notice that B\ can be empty. Further define 1\ so that 

C n EBUB 1 iff n G A. (3.21) 

(a) Suppose ^ 7^ 0. Let 

B := {Cl n E h} U {B U B 1 \ {C n , n E h}}. (3.22) 

Denote 

A B " = HeBnB Wi} U {U ieBl nB W t \ AA. 

Then the subsystem (X^ B °(t), % E A B °) is ergodic, since it is completely 
connected, and (3.9) holds for any x,y E A B °, x 7^ y. Also, we can compute 
as in (3.13) 

tt> = y-^-. — , (3.23) 

Taking into account that > 1, we easily derive from (3.23) the following 
upper bound for the components of the second vector field v^ B ° : 

„ r >_ 1 + £±i^_zI2> (3.24) 
z ~ a + {\A B °\ - 1)6 v ' 
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for any z G A\A B ° due to condition (3.5). Hence, (Xf-(t), i G A) is transient 
according to Theorem A. This contradicts our assumption that A is a trap. 

(b) Suppose now that ii = 0. Then it follows from (3.21) and (3.17) that 
we also have 

/' = 0. (3.25) 

This together with our assumption (3.18) implies that 7^0. Thus in this 
case we have 

A = {x G A : x G U ieB Ai}, (3.26) 
where B 3 {C n , n G /}, and 

A = {U ieB Wi} U {U i6Bl Wi \ Ai}, (3.27) 

where {SUSi}nC n < 1 for any n, according to (3.21) and (3.25). The latter 
implies that (3.9) holds for any y G A which in turn implies ergodicity of 
(Xf(t), % G A). Furthermore, we can find 7r^, analogously to (3.13), namely: 

vrf = — , x G A. (3.28) 

a + (\A\-l)b 

Let us compute now the xth component of the second vector field v% for 
x G Ai with % G I. According to (3.28) and assumption (3.4) we get 

^=- 1 + n+ g- 1)t <°- < 3 ^» 

The latter contradicts our assumption, that A is a trap. 

Hence we conclude that if A is a trap then necessarily condition (3.8) 
holds. Assume now situation (II). More precisely, suppose that A satisfies 
(3.8) but does not satisfy (3.7), i.e. 

A = {x G A : x G U^sAi}, (3.30) 

for some B C A such that (3.8) holds, while Wi \ Ai ^ for i G B x C B for 
some nonempty B\ (see definition (3.20) ). The latter implies 

A = {U ieB Wi} U {U ieBl Wi \ Ai}, 

where according to the assumption (3.8) 

| Si n C n \ = 1 (3.31) 
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for at least one C n . The latter implies C n G B U -Bi, i.e. the set I\ defined 
in (3.21) is non-empty. But as we have seen, this situation contradicts the 
assumption that A is a trap. This finishes our argument that conditions (3.7) 
and (3.8) are necessary for a set A to be a trap. This completes the proof of 
Theorem 3. 

3.3 Learning rule for the almost symmetric case 

We will show here that the Hebbian rule (see [5]) of learning patterns which 
successfully works in the case of Hopfield networks ( [5] ) , is applicable for our 
network at least for the following particular case. 

For any A C A define configuration £(A) = (&{A), i G A) G {-1,+1} A 
such that 

~ { -1, otherwise. (3.32) 

We shall also call a configuration £ a trap for X(t) if and only if £ = £(A), 
where A is a trap. 

Suppose that we are given 2 P , p — binary vectors (images) £i, . . . , £ 2P . 
We shall find the connection constants ~E9 xy such that X(t) satisfying (3.3) 
with these parameters, possesses a system of traps consisting exactly of the 
given 2 P vectors. 

Notice the difference between this task and the problem of stability of 
patterns for Hopfield neural model (see for example, [9] for a recent account 
on the relevant results). Recall, that the capacity of Hopfield network is 
determined by the number of given i.i.d. patterns, which are stable with 
respect to the dynamics of the system. This means that starting from an 
arbitrary initial state the system should converge with a large probability 
to one of the given patterns, which is the closest to the initial state. It 
was conjectured that the number of such patterns for Hopfield network of 
N neurons is at most a fraction of N. Here we construct a network which 
posesses given exponentially large (with respect to N) number of the limiting 
patterns and only them. These patterns are stable in a trivial sense, i.e. if 
the initial state of the system is one of the given traps (patterns), then the 
system stays at this trap forever. However, we do not predict which state 
(out of p possible) the system converges to, starting from an arbitrary initial 
condition. The problem of determination of the basins of attraction of the 
limiting patterns for the hourglass model will be a subject of a separate study. 
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Corollary 1 Suppose the collection of N- dim vectors = 1,...,M} 

where M = 2 P , and £g G {—1, 1} for any /i and 1 < x < N , has the following 
properties: 

!■ ixiy = 1 f or an V fiifx,yeW n for some n G {1, ...,2p}, 
2- £li££ = /or oW/i, 

5. /or any n G {1, ...,2p} i/iere exzsfe unique I = l(n) G {1, ...,2p} \ {n} 
snc/i £/iai = — 1 for any \i whenever x G W n and y G W\. 

Then the N -neuron system with at = a and the connection constants 
E6 = { b ( x ' y ^ ifK x ,y)= min {x',y')H x ',y') (333) 



max( x ^/) o(x', y'), otherwise, 



where 



i M 

&(*,!/) :=Aa— " ^ ( 3 - 34 ) 



(3.35) 



■urai/i £/ie constants A and B satisfying the conditions 

< B — A < 1, 
1< B + A, 

has a system of traps, which is /i — 1, . . . , 2 P }. 

Proof. Indeed, having conditions 1-5* of the corollary satisfied, we derive 
from (3.34) that 

b(x,y) = l-(i + B J°> l** W ™V* W m> (3.36) 

V ,yJ \ —(B — A)a, ifx,yeW n , v ; 

and 

-(A + B)a < b(x, y) <-{B - A) a otherwise. (3.37) 

Substituting (3.36) snd (3.37) into (3.33), and taking into account condition 
(3.35), we obtain 

-E9 xy = (A + B)a > a, if x G W n , y G W i{n)) _ 3g , 

—E6 xy = (B — A) a < a, otherwise, 

which shows that conditions (3.4)-(3.5) are satisfied. Hence we can use Theo- 
rem 3 to conclude that fj, — 1, . . . , 2 P } is a system of traps for the defined 
network. 

Acknowledgements We thank the referee for useful remarks. 
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